MEMORANDUM January 30, 2017 


TO: Lance Menster 
Officer, Elementary Curriculum and Development 


FROM: Carla Stevens 
Assistant Superintendent, Research and Accountability 


SUBJECT: THE EFFECT OF THE WRITER’S WORKSHOP ON THE EXPOSITORY 
WRITING PERFORMANCE OF THE HOUSTON INDEPENDENT SCHOOL 
DISTRICT’S FOURTH-GRADE STUDENTS, 2015-2016 


Writer’s workshop is a process approach to writing instruction using a workshop model being 
implemented in the Houston Independent School District's kindergarten to fourth grade. The 
process involves prewriting or planning, drafting, revision, editing, and evaluation at students’ 
pace and, in some cases, the publication of writing pieces using a variety of genres, ideas, 
organizations, and tones. 


The purpose of this evaluation was to determine the effects of the Writer’s Workshop on 
teachers’ preparation for writing instruction, their actual writing instruction, and on students’ 
expository writing practices using teacher surveys, classroom observations, and treatment 
effects analysis of students’ writing performance. 


Key findings include: 


e According to 2015—2016 eTrain data, between 48.0 and 65.7 percent of teachers in three 
cohorts, who enrolled in the Writer's Workshop professional development (PD), completed 
the training. 

e Fifty percent of 57 teachers who responded to the survey indicated that they dedicated 30 
minutes to daily instructional writing. 

e With an average rating of 3.71 on a 5-point scale, teacher respondents agreed or somewhat 
agreed that the HISD curriculum resources were appropriate and effective for implementing 
Writer's Workshop. 

e Six of the eleven Writer's Workshop classrooms observed in April 2016 had 66.7 to 83.3 
percent of the 24 components on the observation schedule based on one observer. 

e The treatment effect with regression adjustment analysis showed that fourth-grade students 
whose teachers completed the Writer's Workshop professional development had an 
average raw score that was 0.22 items lower on the expository component of the 2016 
STAAR writing test compared to the average raw score of students whose teacher did not 
complete the PD. The difference was statistically significant. 


Further distribution of this report is at your discretion. Should you have any further questions, 
please contact me at 713-556-6700. 
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The Effect of the Writer’s Workshop on the Expository Writing 
Performance of the Houston Independent School District’s 
Fourth-Grade Students, 2015—2016 


Executive Summary 


The Writer’s Workshop is an approach to writing instruction using a workshop model. It is being implemented 
in kindergarten to fourth-grade in the Houston Independent School District. The process involves prewriting 
or planning, drafting, revision, editing, and evaluation at the students’ pace and, in some cases, publishing 
of writing pieces using a variety of topics, genres, ideas, organizations, and tones. The Writer’s Workshop 
sought to improve students’ writing through teacher preparation by accommodating weaker students in 
research-based approaches that treat writing as a process rather than a product (Murray, November, 1972). 
Writer's Workshop treats students as bona-fide authors, exposing them to the approaches authors use in 
their writing. Teachers were exposed to four days and/or 12 hours of training in the writing process in 
preparation for instructing students using Writer's Workshop. Writing is seen as essential to effective 
communication, learning, and self-expression (Graham, Bollinger, Olson, D'Aoust, MacArthur, McCuthen & 
Longhouse, 2012). 


The purpose of this evaluation was to determine the effects of the Writers Workshop on teachers’ 
preparation for writing instruction, their writing instruction, and on students’ expository writing practices and 
performance. The effect of the workshop on students’ writing performance was determined using the State 
of Texas Assessments of Academic Readiness (STAAR) writing scale and raw scores and the percentage 
of students who met the state’s writing standards, a teacher survey, and classroom observations. 


Highlights 


e According to 2015-2016 eTrain data, between 48.0 and 65.7 percent of teachers in three cohorts, who 
enrolled in the Writer's Workshop professional development (PD), completed the training. 


e Fifty percent of 56 teachers who responded to the survey question indicated that they dedicated 30 
minutes to daily instructional writing. 


e The teachers who were surveyed had an overall rating of 1.70 on a 2-point scale with “1” being no and 
“2” being “yes” for their training and preparation for implementing Writer’s Workshop. With an average 
rating of 3.71 on a 5-point scale, teacher respondents agreed or somewhat agreed that the HISD 
curriculum resources were appropriate and effective for implementing Writer's Workshop. 


e Ona 5-point scale, teachers who were surveyed had an overall rating of 3.76 on a 5-point scale 
indicating that they agreed or somewhat agreed that they were efficacious and that they adhered to the 
writing instructional practices. 


e The teachers who were surveyed either somewhat agreed or agreed that their students, 3.49 on a 5- 
point scale, were able to practice writing using the fundamental Writers Workshop principles and 
components. 


e = Six of the eleven Writer’s Workshop classrooms observed in April 2016 had 66.7 to 83.3 percent of the 
24 components on the observation schedule based on one observer. 
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A comparable portion of fourth-grade students in the group whose teachers completed the Writer’s 
Workshop training (61.5%) and whose teachers did not complete the training (60.7%) met the Level II 
Satisfactory at the 2016 progression standard on the STAAR 2016 writing test. 


Using a treatment effect with regression adjustment analysis, fourth-grade students whose teachers 
completed the Writers Workshop PD had an average scale score on the 2016 STAAR writing test that 
was 104 points lower than the scale score of students whose teachers did not complete the PD. The 
difference was statistically significant. 


The treatment effect with regression adjustment analysis also showed that fourth-grade students whose 
teachers completed the Writer's Workshop professional development had an average raw score that 
was 0.22 items lower on the expository component of the 2016 STAAR writing test compared to the 
average raw score of students whose teacher did not complete the PD. The difference was statistically 
significant. 


Recommendations 


Teacher completion of the professional development designed to enhance preparation for effective 
delivery of the Writer's Workshop needs to be encouraged. The high contract to participation cost 
incurred in delivering this professional development and the failure of teacher participants to confidently 
confirm students’ ability to outline the writing process using the Bomer’s Flowchart makes the case for 
the encouragement. 


Teachers may require additional resources to effectively teach writing at the elementary school level. It 
may be essential to ensure that writing receives the same level of attention at all elementary grade levels 
in addition to the fourth grade where writing is tested. 


Future evaluations should focus on observing the delivery of the Writer’s Workshop, particularly before 
the testing period, and possibly multiple times during the school year to assess teachers’ actual writing 
instructional practices. This would allow for a more reliable observation of the Writer’s Workshop and a 
more robust assessment of fidelity. 
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Introduction 


The Houston Independent School District (HISD) implemented the Writer’s Workshop in kindergarten to 
eighth grades in all schools during the 2015-2016 school year. The Writer’s Workshop uses the process 
approach to writing instruction within a workshop model. The process involves prewriting or planning, 
drafting, revising, editing, evaluating at students’ pace, and in some cases the publication of written pieces 
using a variety of topics, genres, ideas, organization, and tones. Students are seen as bona-fide authors 
and, therefore, are exposed to the approaches authors use in their writing. In preparation for facilitating 
students’ writing, teachers of writing were exposed to a four-day and/or 12-hour training in the writing process 
or Writer's Workshop. 


Writing has been described as an essential aspect of “professional, social, community, and civic” 
engagement because the ability to write well is fundamental to effective “communication, learning, and self- 
expression” (Graham, et al., 2012, p. 6). A national survey of elementary school teaching instruction, grades 
4 — 6, found that teachers were inadequately prepared to teach writing, students’ time devoted to writing was 
inadequate, and that students’ writings were commonly restricted to narratives, letter writing, completing 
worksheets, and responding to materials read (Gilbert & Graham, 2010). There was little evidence that 
teachers used research-based approaches when teaching writing, and that they made few or no adaptations 
or did so infrequently to facilitate weaker students (Gilbert & Graham, 2010). On the 2007 National 
Assessment of Educational Progress (NAEP) writing assessment, 67 percent of eighth-grade students 
performed at or below the basic level in writing indicating that they had attained “only partial mastery of the 
writing skills needed’ at this grade level (National Assessment of Educational Progress, Salahu-Din, Persky 
& Miller, 2008 cited in Gilbert & Graham, 2010). Sixty-two and 63 percent of HISD fourth- and seventh-grade 
students, respectively met phase-in 1 Level Il: Satisfactory standard on the 2015 State of Texas 
Assessments of Academic Readiness (STAAR) writing test, compared to 70 and 72 percent respectively at 
the state level. STAAR writing in HISD and the State of Texas is tested in the fourth and seventh grades. 


The Writers Workshop seeks to improve students’ writing through teacher preparation, use of a variety of 
genres and topics, and use of research-based approaches that treat writing as a “process rather than a 
product” (Murray, 1972, p. 3). The purpose of this evaluation, therefore, is to determine the effects of the 
writing workshop on teacher preparation for teaching writing, writing instruction, and students’ expository 
writing practices and performance. The effect of the workshop will be determined using STAAR writing 
scores and performances and Stata treatment effects with regression adjustments (teffects ra), a teachers’ 
survey, and classroom observations. The evaluation sought to answer the following four questions: 


1. What were the perceptions and experiences of teachers who participated in the Writer’s Workshop? 


2. What were the demographic characteristics of the Writer's Workshop fourth-grade student sample? 


3. How did fourth-grade students whose teachers participated in the Writer’s Workshop perform on the 
overall and expository component of the 2016 STAAR writing assessment? 


4. What was the effect of the Writer's Workshop instruction on the overall and expository writing 
performance of fourth-grade students whose teachers completed the Writer’s Workshop? 
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Literature Review 


Expository writing involves communicating information about a topic using facts and details. It includes 
specific terms and materials that help classify and define the topic. This type of writing is organized using 
structures such as examples, classification, cause and effect, problem and solution, and compare and 
contrast (Houston Independent School District, 2015; Purdue University, 2016). The narrative and expository 
writing samples of sixth- to twelfth-grade students with language learning disabilities were evaluated for 
elements of microstructure? and macrostructure? using experimentation. Results of the Wilcoxon signed 
ranks tests indicated that the narrative genre had significantly higher levels of productivity and grammatical 
complexity compared to the expository genres. Both genres showed impoverished text structure and 
revealed strong correlations between the microstructure and macrostructure (Hall-Mills & Apel, 2013). A 
report on a 2008 national writing survey concluded that despite consensus on research-based approaches 
to writing, they have not been widely implemented (Laud, 2013). In addition, most schools have insufficient 
comprehensive, sustained, and focused systems for offering teacher professional development to support 
research-based writing practices (Laud, 2013). 


Research on the effects of the Writer's Workshop has been mixed. A study involving 90 kindergarten 
students, randomly assigned to journal writing and Writer's Workshop classrooms, revealed a significant 
difference in students’ writing achievement in favor of those who were exposed to the Writer’s Workshop 
strategies using a repeated measures test on a 10-stage developmental writing rubric (Mester, 2011). When 
a Chi-square contingency table was used to determine the effect of the Writer’s Workshop on 321 students 
in achievement on the Georgia fifth grade writing assessment over three years, the results showed that 
students’ initial performance prior to the intervention was the highest (Smithson, 2008). In addition, 
independent samples t-test of the latter two years’ performance revealed no significant scale score difference 
between the groups’ writing performance on the assessment (Smithson, 2008). Smithson (2008) identified 
at least three factors that may explain the unexpected findings of the Georgia study: (1) teacher resistance 
to full implementation of Writers Workshop due in part to inadequate training and teacher preparation, (2) 
changes in the test ratings and achievement standards, and (3) lower student ability over time. 


The writing performance of 18 first-grade students was compared using writing prompts and Writer’s 
Workshop strategies during a five-week period. The result showed that the reading attitudes of both groups 
declined while the group exposed to the prompts demonstrated a larger decline. Students who were taught 
using prompts, however, had mean scores that were higher than those who used the Writer’s Workshop 
strategies. Overall, students who were better writers enjoyed the writing process more (Caroll & Feng, 2010). 
This study, however, did not specifically investigate Writer’s Workshop’s effect on expository writing. 


The United States Department of Education Institute of Education Sciences (IES) outlined four 
recommendations for teaching elementary school students to be effective writers (Graham, Bolinger, Olson, 
D’Aoust, MacArthur, McCuthen & Longhouse, 2012). These were (1) provide daily time for students to 
write; (2) teach students to use the writing process and how to write for a variety of purposes; (3) teach 
students to become fluent with handwriting, spelling, sentence construction, typing, and word processing; 
and (4) create an engaged community of writers. (Graham, et al., 2012). Writer’s Workshop, as an approach 
to the writing process, satisfies the Institute’s recommendations for teaching elementary students to be 
effective writers. Writer's Workshop’s effectiveness is the focus of this evaluation study. 


1 Microstructure involves the text base, writer's conveyance of meaning at multiple level (word, sentence), productivity, grammatical complexity and 
accuracy, and lexical diversity 

? These include genre-specific text structure elements like the “gist of the text, organization, cohesion, and text structure, writer’s conveyance of 
meaning. 
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Methods 


Data Collection and Analysis 


Data collection for the evaluation of the Writers Workshop included an online survey, classroom 
observations, and analysis of students’ overall and expository writing results on the 2016 STAAR fourth- 
grade writing assessment. STAAR writing in elementary school is tested only in the fourth grade. The 
evaluation benefited from several meetings and discussions with the elementary curriculum officer and 
managers responsible for supervising the implementation of the Writer’s Workshop including the 
professional development (PD) component. It also involved the review of the PD documents and other online 
document resources related to writing and the Writer's Workshop. 


An online questionnaire was administered to teachers who completed the Writer's Workshop PD training. 
The survey questions were developed based on the course content. The questionnaire was vetted for validity 
by the elementary curriculum officer and managers, which they disseminated to increase teachers’ response 
rate. The survey was administered using SurveyMonkey™. SurveyMonkey™ is a free web-based software 
that administers, collates, and analyzes survey data. The survey was administered from May 12 through 
27, 2016. SurveyMonkey™ calculated an average rating scale score (See Appendix A on p. 19 for details). 
The survey results constitute a major part of this research report. 


A sample of schools were randomly selected for classroom observations based on their performance on the 
2014-2015 STAAR writing test. Schools were ranked according to the percent of students who met Level II: 
Phase-in 1 Satisfactory standard and stratified into deciles. One school from each decile stratum was 
selected by random resulting in a total of ten schools. Emails were sent to the principals of these schools 
explaining the purpose and procedures for the visit. Six schools agreed to the visit. Classroom observations 
were conducted during regular fourth-grade writing instruction classes. Teachers were informed of the visit 
beforehand. 


All visits were scheduled between April 11 and 22, 2016 based on the English language arts (ELA) Pacing 
Calendar? to coincide closely with expository writing instruction and to accommodate full implementation of 
the Writer's Workshop and STAAR testing. However, two of these visits had to be rescheduled and were 
held April 27 and May 13, 2016. An observation schedule was developed using information from the “Writer's 
Workshop Facilitator's Guide” and “The First 25 Days of Writer's Workshop” implementation document. 
Principals or their representatives were debriefed following each school visit to discuss observations and 
possible recommendations for improvements in program implementation. The results of the school visits 
were used to determine the program’s apparent’ fidelity. Fidelity has been described as “the proportion of a 
program’s components that was implemented” (Mowbray, Holter, Teague & Deborah, 2003, p. 316). 


Teachers serving students from kindergarten through fifth grade and who attended the summer Writer's 
Workshop training in June and July 2015 were included in this study. They were selected because their 
students, particularly in the fourth grade, would have had a full year’s exposure to the teacher’s writing 
instruction skills and expertise acquired during the Writer’s Workshop PD training. Other training sessions 
were held in September, October, and November 2015, and January 2016 for kindergarten to second grade 
and third to fifth-grade teacher cohorts. 


3 The ELA Pacing Calendar is a general guide prepared by HISD Curriculum and Development Department for the scheduled delivery of key Texas 
Essential Knowledge and Skills (TEKS) and ELA curriculum components. 

4 | described it as apparent because | was unable to actually observe Writer's Workshop lesson delivery in a number of cases since observation 
occurred post-assessment when active delivery of the writing process had ended. 
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Participation in the Writer’s Workshop PD training was voluntary or principals could have recommended 
teachers. Only teachers who registered for the PD, however, were included in this study. This study, 
therefore, is restricted to teachers who either completed or were recorded as “no show’ in the Writer's 
Workshop PD sessions. Teachers and their students who completed the PD comprised the treatment group 
and students and their teachers who were “no show” made up the non-treatment group. 


Students’ raw scores (number of items correct) on the 2016 STAAR fourth-grade expository writing 
component and their overall writing scale scores were used to analyze students’ performance and determine 
program effect. Students’ overall writing scale scores and performance data were also used to determine 
the mean scale score difference between students whose teachers completed the program and teachers 
who were recorded as “no show’ in the program. Analyses were based on the number of fourth-grade 
students who met the 2016 STAAR Level Il: Satisfactory and Level Ill: Advanced writing performance at the 
2016 progression standard. 


Sample 


The sample consisted of students whose teachers completed the Writer's Workshop PD training and those 
whose teachers were recorded as ‘no show’ and whose students scored on the fourth-grade STAAR writing 
test. A total of 892 teachers registered to participate in the professional development program. These 
participants and their writing students constituted the “frame” from which the fourth-grade student sample 
was selected (see Murnane & Willet, 2011, p. 47). Generalization of results are limited, therefore, to the 
broader group of fourth-grade students whose teachers registered to participate in the program and not to 
the total population of fourth-grade students (see Murnane & Willet, 2011, p. 47). Forty-three fourth-grade 
teachers completed the PD and 12 fourth-grade teachers were recorded as “no-show”, making up the fourth- 
grade teacher sample. 


Students were linked to their fourth-grade teachers using the Public Education Information Management 
System (PEIMS) and the STAAR assessment result files. These files included demographic and test data 
for each student. A total of 2,371 fourth-grade students comprised the sample. Of these, 1,879 students had 
teachers who completed the Writer's Workshop training and 492 students had teachers who were recorded 
as “no show” in the program. Tests of normality using detrended normal Q-Q and normal Q-Q plot of the 
scale score and the raw score for the composition (expository) component of the STAAR test were 
confirmed. 


Of the 447 teachers who completed the PD, 57 (12.8%) completed the survey questionnaire. As mentioned 
earlier, 11 writing classes in six of the 10 randomly selected schools were observed. A rubric of Writer’s 
Workshop criteria was used to conduct the evaluation by one observer (See Appendix B, p. 20). 


Data Limitations 


e As noted, the STAAR writing assessment is limited to fourth grade and, therefore, does not cover the 
full spectrum of grades whose teachers participated in the Writer's Workshop. 


e Because of the limited survey administration timeframe, only 12.8 percent of teachers responded to the 
Writer's Workshop survey. Any inference about the survey has to be restricted to that group of teachers 
since they may not be representative of the entire population of Writers Workshop participants in HISD. 


e Classroom observations were conducted toward the end of the school year to minimize disruption. 


Instruction had to be scheduled to accommodate the visits and may not have reflected the preferred 
natural setting for observation. 
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Classroom observations were conducted after the STAAR testing in April 2016 by which time most 
manipulatives and Writer's Workshop artifacts were removed from the classroom to ensure test security. 
The observations, therefore may not be a true reflection of the students’ entire writing experience since 
it captures a moment in time. Results of the observations, may not adequately reflect program fidelity. 


Results 


What were the perceptions and experiences of teachers who participated in the Writer’s Workshop? 


Teacher Participation 


The Writers Workshop PD was offered to kindergarten through fifth-grade teachers of writing. Summer 
sessions were held in June and July, 2015 for these teachers. As mentioned before, additional sessions 
were held in September, October, and November 2015, and January 2016 for third- to fifth- and kindergarten 
to fifth-grade teachers. A total of 892 teachers enrolled in the PD, voluntarily. Of these, 611 were enrolled in 
the summer’s kindergarten through fifth-grade training; 108 teachers enrolled in the third to fifth grade PD 
training, and 173 teachers were enrolled in the kindergarten to second grade PD training. Of the 892 
teachers who enrolled in the Writers Workshop PD, 447 teachers completed the PD, 212 were “no shows”, 
79 did not complete the PD, and 154 teachers dropped out of the PD. Figure 1 shows the participation status 
of teachers by the Writer's Workshop PD cohorts. 


Figure 1. Teachers’ participation status by Writer’s Workshop professional development cohorts, 


2015-2016 
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Figure 1 shows that 65.7 percent of third to fifth grade teachers competed the Summer Writer’s 
Workshop PD. 


Less than half of the teachers who enrolled in the kindergarten to fifth grade Writer's Workshop PD 
completed the training. 


Between 20.4 and 28.9 percent of teachers enrolled in the Writer's Workshop PD training cohorts were 
“no shows.” 
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e Of the teachers who enrolled in the kindergarten to fifth and kindergarten to second-grade Writer’s 
Workshop PD cohort, 18.8 and 17.9 percent, respectively, dropped out. 


Survey Results: 
a. Instructional Contexts for Writer’s Workshop 


An online survey was administered to teachers who completed the Writer's Workshop PD training. Fifty- 
seven prekindergarten to fifth-grade teachers (12.8% of those who completed the PD) completed the survey 
as shown in Figure 2. 


Figure 2. Distribution of Writer’s Workshop teacher survey respondents by grade (n = 57) 
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Note: Teachers in departmentalized schools teach writing in multiple grades. There were 57 respondents who taught 66 writing classes. 


e Most of the survey respondents were first-grade (26.3%) and second-grade (26.3%) teachers. 
e According to Figure 2, 15.8 percent of respondents were third and fourth-grade teachers. 


e Survey respondents were asked to identify the teaching configuration in which they implemented the 
Writer's Workshop. Of the 57 respondents, 50.9 percent said it was delivered in self-contained classes, 
40.4 percent said it was delivered in departmentalized classrooms, and 8.8 percent said they delivered 
the Writers Workshop in other configurations that included, dual language, team teaching, special 
education, and intervention settings. 


e Most respondents provided writing instructions in English (55.4%) as opposed to Spanish (42.9%) and 
Chinese Mandarin (1.8%). Figure 3 shows the approximate number of minutes teachers devoted to 
daily writing instructions. The suggested duration of daily writing instruction is 45-60 minutes (Bach & 
Supovitz, 2003). 
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Figure 3. Approximate number of minutes dedicated to daily writing instruction by Writer’s 
Workshop survey respondents (n = 56) 
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e Most instruction in writing appeared to last about 30 minutes according to half of the survey respondents 
(50.0%). 


e About one-third of the survey respondents (30.4%) dedicated approximately 45 minutes to daily 
instruction in writing. 


e Almost one-fifth (19.6%) of survey respondents devoted approximately 60 minutes for daily writing 
instruction. 


b. Preparation and Training 
A five-item Likert scale was used to measure teachers’ preparation and training for delivering Writer's 


Workshop writing instructions. Items were weighted and average ratings were tabulated. Fifty-three (93.0%) 
of teacher respondents responded to the statements shown in Table 1. 


Table 1. Teacher preparation and training of implementing Writer’s Workshop 


Preparation and training No Yes Average 
(1) (2) Rating 
n n 
Attended four-day Writer’s Workshop session 10 39 1.80 
Attended the 12-hour Writer's Workshop session 26 21 1.42 
Information learned in the four-day sessions prepared me for Writers’ Workshop. 3 48 1.94 
The read-aloud books worked well with the writing process. 2 50 1.96 
The “First 25-days” document provided adequate guide for teaching writing. 3 50 1.94 
Overall Rating 1.70 


e =With an overall rating of 1.70 on a 2-point scale with “no” being 1 and “yes” being 2, teacher respondents 
indicated that they were adequately prepared for implementing the Writer's Workshop in their 
classrooms. Overall, most respondents met the preparation and training requirements outlined. 


e Most teacher respondents indicated that (a) the read-aloud books used for the writing process worked 
well (average rating of 1.96), (b) information learned in the four-day session prepared them for Writer's 
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Workshop (average rating of 1.94), and (c) the guidance provided by the “First 25-days document” 
appeared to have been an adequate guide for teaching writing (average rating of 1.94). 


e More respondents appeared to have attended the four days Writer's Workshop session than those who 
attended the 12-hour Writer's Workshop session. Their average ratings was 1.80 and 1.42, respectively, 
on a 2-point scale. Some may have attended both sessions. 


Teacher respondents were also asked to suggest any additional PD that they would like to receive in support 
of the Writer’s Workshop implementation. Twenty-one respondents (36.8% of all respondents) provided 
suggestions. The four most frequent suggestions were: ongoing writing PD workshops throughout the year 
(5 respondents); strategies for implementing Writer's Workshop in kindergarten (3 respondents); improved 
time management (2 respondents), and the use of writing stations (2 respondents). 


c. Appropriateness and Effectiveness of HISD Curricular Resources for Writer’s Workshop 


Respondents’ levels of agreement or disagreement with six statements were used to measure the 
appropriateness and effectiveness of related district resources for supporting the Writer’s Workshop. The 
items were rated on a scale of 1 to 5, from strongly disagree to strongly agree. The results are summarized 
in Table 2. 


Table 2. Appropriateness and effectiveness of HISD curricular resources for Writer's Workshop 


implementation 


HISR Curricular Resources Strongly Disagree Somewhat Agree Strongly Average 
Disagree Agree Agree Rating 
(1) (2) (3) (4) (5) 
n n n n n 


Structure and framework of HISD 

Curriculum and Planning guide facilitate 1 4 18 25 5 3.55 
ease of planning. 

First 25-days document supports Writer’s 

Workshop implementation. 1 2 12 26 12 3.87 


Sixty-minutes writing session provided 
structure to classroom writing. 2 5 15 26 4 3.48 


Instructional strategies in HISD curriculum 

planning guides supported my 1 4 8 33 5 3.73 
instructional planning and delivery. 

Instructional strategies in HISD curriculum 

guides supported my _ instructional 1 4 11 32 5 3.68 
planning and delivery. 


Anchor charts in HISD curriculum planning 


guides supported my _ instructional { 4 B 33 12 3.96 
planning and delivery. 
Overall Rating 3.71 


e The appropriateness and effectiveness of HISD curricular resources in the planning and delivery of the 
Writer's Workshop had an overall rating of 3.71 on a 5-point scale. Most respondents agreed with the 
statements in Table 2. 


e With an average rating of 3.96 on a 5-point scale, most respondents agreed or strongly agreed that the 
anchor charts appeared to have supported their instructional planning and delivery. 


e §=6The “First 25-days document” had an average rating of 3.87 on a 5-point scale, as a support for the 
implementation of the Writer’s Workshop. Most respondents agreed with this statement. 
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e Onan average rating of 3.55 on a 5-point scale, most respondents either agreed or somewhat agreed 
that the structure and framework of HISD curriculum and planning guides facilitated the ease of planning 
for Writer's Workshop. 


e The lowest average rating of 3.48 on a 5-point scale was received on “the 60-minutes writing sessions 
provided structure to writing in the classroom”. Most respondents either agreed or somewhat agreed 
with this statement. 


Teacher respondents were also asked to provide additional details or information that could be included in 
the curriculum planning guides to support the Writer's Workshop implementation. Fourteen respondents 
provided details and information that included: linking the Writer's Workshop to exemplar lesson plans and 
teaching strategies; aligning Writer's Workshop to existing district resources; including teachers in the 
preparation of the curriculum guides; organizing the curriculum better to follow a logical sequence; 
condensing the planning guides; sticking to the instruction time frames; and offering actual writing topics. 


d. Teacher Self-efficacy and Writing Instructional Practices 


Seven survey items were used to measure teacher self-efficacy and writing instructional practices in the 
classroom. The items measured the extent of teacher agreement or disagreement on a 5-point Likert scale 
as shown in Table 3. 


Table 3. Survey respondents’ self-efficacy and writing instructional practices 
Self-Efficacy and Writing Instructional Strongly Disagree Somewhat Agree Strongly Average 
Practices Disagree Agree Agree Rating 


(1) (2) (3) (4) (5) 
n n n 


Compared to a year ago, | have a deep 
understanding of what it takes to be an 0 4 9 27 12 3.90 
effective writing teacher. 

| successfully guide my students to identify 


writing topics. g = S Se e Se 
| facilitate small group oral story crafting in 2 8 12 19 11 356 
my classroom. ; 

| use rubrics to assess my students’ writing 0 4 14 24 10 3.77 
I used read aloud daily as part of my writing 0 3 13 22 14 3.90 
instruction. 

| use 6+1Traits of Writing in my classroom 

daily for common_ language and 2 6 24 11 8 3.33 
demonstration of good writing. 

Identified English/Spanish Language Arts 

TEKS are practiced as a common thread to 0 3 9 28 12 3.94 
my daily writing instruction. 

Overall Rating 3.76 


e With an overall rating of 3.76 on a 5-point scale, most teacher respondents agreed or somewhat agreed 
with the statements related to their self-efficacy and writing instructional practices. 


e “Identified English/Spanish Language Arts TEKS are practiced as a common thread to my daily writing 
instruction” and “I successfully guide students to identify writing topics” received the highest average 
ratings of 3.94 and 3.92, respectively, on a 5-point scale. Most respondents either agreed or strongly 
agreed with these statements. 
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e Most respondents either agreed or strongly agreed that “I used read aloud daily as part of my writing 
instruction” and that “compared to a year ago, | have a deep understanding of what it takes to be an 
effective writing teacher.” Both statements had an average rating of 3.90 on a 5-point scale. 


e Respondents use of 6+1 Traits of Writing to create a common language of effective writing and to 
demonstrate what good writing looks like daily, had the lowest rating of 3.33 on a 5-point scale. Most 
respondents somewhat agreed with this statement. 

e The facilitation of small group oral story crafting had an average rating of 3.56 on a 5-point scale. 

e. Students Writing Practices 

Finally, teacher respondents were asked to rate their students’ writing practices using a scale of 1 to 5 to 


measure their degree of agreement or disagreement on eight items. Fifty-one teachers responded to these 
items. The results are shown in Table 4. 


Table 4. Teachers’ ratings of their students’ writing practices 


HISR Curricular Resources Strongly Disagree Somewhat Agree Strongly Average 
Disagree Agree Agree Rating 
(1) (2) (3) (4) (5) 
n n n n n 
My students can outline the writing process 
using Bomer’s Flowchart. 1 17 20 10 3 2.94 
My students are able to make My Heart 
maps as part of the writing process. 0 7 14 18 12 3.69 
My students are able to complete their 
writing station activities daily and on time. 0 7 16 19 9 3.59 
My students are able to use Mentors’ texts, 
appropriately. 0 5 24 14 7 3.46 
My students are able to draft, revise, and 
edit an original piece of writing. 0 6 14 25 6 3.61 
My students confer and reflect on their 
writing pieces. 0 6 19 20 6 3.51 
My students are able to use objects to 
generate their writing. 1 6 13 23 7 3.58 


My students are able to effectively 
incorporate the Writer's Notebook into the 
writing process. 0 iG 18 19 7 3.51 
Overall Rating 3.49 


e Overall, teachers’ average rating of students writing practices was a 3.49 on a 5-point Likert scale, which 
was the lowest of all three Writer's Workshop PD constructs measured on the 5-point Likert scale. The 
average rating for six of the eight items were above the overall rating. Most respondents either agreed 
or somewhat agreed with these statements. 


e “My students are able to make My Heart maps as part of the writing process” received the highest 
average rating of 3.69 on a 5-point scale. This was followed by “My students are able to draft, revise, 
and edit original writing pieces”, which received an average rating of 3.61 on a 5-point scale. Most 
teachers agreed or somewhat agreed with both statements. 


e “My students can outline the writing process using Bomer’s Flowchart” received the lowest average 
rating of 2.94 on a 5-point scale. Most teacher respondents disagreed and somewhat agreed. This was 
followed by students’ ability to use mentor text appropriately, which received an average rating of 3.46 
on a 5-point scale. Most teachers agreed or somewhat agreed with this statement. 
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Teacher respondents were asked to provide additional suggestions with regards to the Writer’s Workshop. 
Ten teachers provided suggestions that included: how to implement Writer's Workshop in kindergarten, more 
mentor text for teachers, review the 6+1 Traits of Writing and Bomer’s Flowchart. One respondent liked the 
idea of the notebook. One respondent highlighted the time constraints in meeting Writer's Workshop 
requirements. 


Writer’s Workshop Classroom Observations 


Twenty-four components of the Writer’s Workshop were selected for observation in 11 classrooms from five 
HISD schools. There was only one observer. Writer’s Workshop ‘apparent fidelity’ (AF) was measured using 
the number of components observed as a percentage of the 24 components on the observation schedule. 
Appendix B (p. 20) shows the AF for each class and across all classes observed. 


e Six (54.5%) of the classrooms had an AF of over 50 percent. Classroom KO4GS had the highest 
apparent fidelity of 87.5 percent, followed by FO4GS with 83.3 percent, and PO4NC with 75.0 percent. 
Classrooms PO5MR and HO4BK had the lowest apparent fidelity of 4.2 and 8.3 percent, respectively. 


e According to Appendix B, between 18.2 and 36.4 percent of the observed classrooms displayed Writer’s 
Workshop listening habits and Heart Maps. Between 45.5 and 63.6 percent used mini-lessons, 
independent writing, draft/writing, proof reading, mentor texts, and demonstrated the links between 
TEKS-writing lessons, and displayed celebration artifacts and writing assessment, progress chart, or 
published students’ work. 


e Appendix B also showed that between 72.7 and 81.8 percent of classrooms visited had writing visuals, 
used writer's notebooks, demonstrated draft/writing, had teacher conferences, and revised and edited 
students’ writing (See Appendix B, p. 20). 


e Exit interviews with elementary teachers and principals revealed that two of the schools also 
implemented the “Writers in the Schools” program to supplement their Writer's Workshop, particularly at 
the lower writing grades. At a cost of about $26,000, the supplemental program focused on narrative 
writing since Writers Workshop focused on expository writing. Exemplary writing pieces from the 
“Writers in the Schools” were published into bound books. One of the two schools was a literature 
magnet. “Writers in the School” provided supplemental resource materials that were used to facilitate 
instruction and practice in the writing process at these schools. 


e Data available for eight classrooms showed that where 66.7 to 87.5 percent of the Writer's Workshop 
prompts were observed, between 52.7 and 77.8 percent of the students met Level II: Satisfactory 
performance at the 2016 progression standard except one classroom with an AF of 70.8 percent where 
29.9 percent of students met Level II: Satisfactory performance. 


e One classroom with an AF of 8.3 percent had 58.5 percent of the students and another with 16.7 percent 
AF had 87.9 percent of the students who met Level II: Satisfactory performance on the STAAR fourth- 
grade writing test. Similar classroom performance patterns were observed for the average number of 
items correct on the expository component of the STAAR fourth-grade writing test. Caution, however, 
must be exercised in the interpretation of the AF since, as part of the limitations, the observations were 
conducted toward the end of the school year and classrooms artifacts were taken down in accordance 
with STAAR testing procedures. 
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e ©The relationship between Writers Workshop AF and the percentage of students who met Level II: 
Satisfactory performance on the 2016 progression standard on the STAAR fourth-grade writing was 
weak and negative (-0.24), the relationship for the expository writing component was moderate and 
negative (-0.54). 


What were the demographic characteristics of the Writer’s Workshop fourth-grade student sample? 


The proportion of students in the treatment and non-treatment groups disaggregated by key demographic 
and educational attributes are presented in Figure 4. As noted earlier, fourth-grade students were linked to 
their teachers who enrolled® in the Writer's Workshop using unique identifiers in PEIMS. These students 
constituted the sample for determining the effects of the Writers Workshop on their 2016 STAAR fourth- 
grade writing performance. 


Figure 4. The demographic and educational composition of the treatment and non-treatment 
groups in the study sample, Fourth Grade, 2015-2016 


100 
90 
80 
70 
a 60 
S 
< 50 
(cr) 
2 
ro) 40 
ja 
30 
20 
10 
9 Female Male No Yes No Yes No Yes No Yes No Yes Black |Hispanic} White 
Gender Econ. Disadv. At Risk Special Ed. LEP G/T Ethnicity 
= Treatment Group (%) 50.6 49.4 14.8 85.2 44.2 55.8 95.1 49 65.2 34.7 80.6 19.4 29.4 64.8 3.2 
= Non-treatment Group (%) 53.6 46.3 17.5 82.5 37.8 62.2 94.9 5.1 59.8 40.2 84.8 15.2 17.1 77.4 4.5 
= District 49.2 50.8 23.5 76.5 36.0 64.0 92.7 7.3 69.8 30.2 85.1 14.9 24.5 62.1 8.4 
= Treatment Group (%) = Non-treatment Group (%) = District 


Note: District data were taken from PEIMS Edit + Reports Data Review, 2015-2016 Fall Collection 


e Proportionally, the largest difference between the treatment and non-treatment groups was among Black 
students(11.9-percentage points) in favor of the treatment group and Hispanic students (12.6- 
percentage points) in favor of the non-treatment group. 


How did fourth-grade students whose teachers participated in the Writer’s Workshop perform on the 
overall and expository component of the 2015-2016 STAAR writing assessment? 


Figure 5 shows the proportion of Writer's Workshop fourth-grade students who met Level II: Satisfactory 
performance and Level Ill: Advanced performance on the STAAR writing test at the 2016 progression 
standard. 


5 This included the treatment group and the non-treatment group, that is, those students whose teachers completed the PD and those whose teachers 
were no show. 
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Figure 5. Fourth-grade STAAR Writing Performance for Students in the Writer’s Workshop Sample, 
HISD, 2015-2016 
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Source: STAAR 3-8 Regular test, July 11, 2016 


e Acomparable portion of fourth-grade students in the observed (61.5%) and comparison (60.7%) study 
groups met Level II: Satisfactory at the 2016 progression standard on the STAAR 2016 Writing test. 


e A lower proportion of fourth-grade students whose teachers completed the Writer's Workshop (11.1%) 
compared to those whose teachers did not participate in the workshop (15.9%) met Level Ill: Advanced 
performance on the 2016 STAAR writing test. 


What was the effect of the Writer’s Workshop instruction on the overall and expository writing 
performance of fourth-grade students whose teachers completed the Writers Workshop? 


The effects of the Writer's Workshop on the STAAR overall and expository writing performance of fourth- 
grade students whose teacher teachers participated in the workshop are presented in Table 5 and Table 6. 
Table 5 shows the effect of the workshop on the overall writing performance using the Stata treatment effect 
with regression adjustment (teffects ra) command regressed on students’ economic status, at-risk status, 
special education, limited English proficiency (LEP), and gifted and talented (G/T) identification. Table 5 
used students’ writing scale scores as the overall outcome and Table 6 used the raw scores for the students’ 
expository writing. Students could have received a total of 8 points on the expository writing. 


Table 5. Average Treatment Effect (ATET) of the Writer’s Workshop instruction on the STAAR writing 


performance of HISD fourth-grade students, 2015-2016 


Writing Scale Coefficient Robust Standard p>z 95% Confidence 
Score Error Interval 
(n = 2,371) 
ATET 
Treatment 
(1 vs. 0) -104.8 28.5 -3.68 .000 -160.6 — -49.0 
POmean 
Treatment 
0 3823.9 27.6 138.5 .000 3769.8 — 3878.1 


Note: The coefficient is the writing scale score. Linear data distribution was assumed for the treatment; POmean is the potential outcome mean® 


e As shown in Table 5, the fourth-grade students whose teachers completed the Writer's Workshop and 
who may have used the workshop’s related instructional practices for writing had an average scale- 


5 The Potential Outcome Mean (POmean) is the mean of the STAAR writing outcomes fourth-grade students would obtain if their teachers were exposed 
to the Writer's Workshop PD training and used the related instructional practices (See http://www.stata.com/manuals13/teglossary.pdf). 
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score that was 104.8 points lower than the potential mean scale score (POmean = 3823.9) of the fourth- 
grade students whose teachers registered for but did not attend the workshop. The difference was 
statistically significant (p>.01) 


Table 6. Average Treatment Effect (ATET) of the Writer’s Workshop instruction on the STAAR expository 


writing performance of HISD fourth-grade students, 2015-2016 


Writing Raw Coefficient Robust Standard Z p>z 95% Confidence 
Score Error Interval 
(n = 2,371) 
ATET 
Treatment 
(1 vs. 0) -.22 .06 -3.43 .0O1 -.34 — -.09 
POmean 
Treatment 
0 3.7 .06 63.1 .000 3.64 - 3.9 


Note. The coefficient are the raw scores on STAAT expository writing marked out of 8 points. Poisson distribution was assumed, therefore, in the calculation 
of the treatment effect; POmean is the potential outcome mean 


e Table 6 shows the fourth-grade students whose teachers completed the Writer's Workshop and who 
may have used the workshop-related instructional practices in writing had an average raw score on their 
STAAR expository writing that was .22 correct items lower than the potential average raw score 
(POmean = 3.7) of their fourth-grade peers whose teachers registered for but did not participate in the 
workshop. 


Discussion 


The purpose of this evaluation was to measure the impact of the Writer's Workshop on the expository writing 
of HISD fourth-grade students, and teachers’ perceptions and experiences with implementing this writing 
process in their HISD classrooms. Because STAAR assesses elementary writing only in the fourth grade, 
the analysis was restricted to that grade. An online survey using Likert scales was used to measure four 
main constructs associated with the Writer’s Workshop: teacher preparation and training for implementing 
the workshop, which received an average rating of 1.7 on a 2-point scale indicating that most respondents 
agreed; the appropriateness and effectiveness of HISD curricular resources for implementing Writers 
Workshop with an average rating of 3.71 on a 5-point scale; teacher self-efficacy and writing instructional 
practices, which received an average rating of 3.76 on a 5-point scale; and teacher ratings of students writing 
practices with an average rating of 3.49 on a 5-point scale. 


Overall, the teachers’ ratings of students’ writing practices had the lowest average rating of 3.49 on a 5-point 
scale. These respondents mostly agreed or somewhat agreed that their students were able to use the key 
elements of the writing process. More respondents somewhat agreed than agreed that their students could 
outline the writing process using Bomer’s Flowchart or use mentor texts, appropriately. This is significant 
since Bomer’s flow chart is a fundamental component of the writing process. The program is designed to get 
students to adopt the attitudes and behaviors of authors but most teacher respondents appeared to disagree 
or somewhat agree that their students could use the flow chart which outlined the writing process to be 
followed. This was the first year of implementation of the program and as such teachers may require more 
time to adopt these writing practices. The Smithson (2008) study conducted in Georgia showed teacher 
resistance to full implementation of the Writer's Workshop because of inadequate training and teacher 
preparation. This study, however, did not ask the teachers in observed classrooms to confirm their 
attendance at the any of the Writers Workshop professional development sessions but the analysis 
attempted to link an apparent fidelity to students’ performance in writing in the classrooms observed. 


Analysis of the relationship between the apparent fidelity and classroom STAAR writing performance 
returned a small but negative correlation (-0.24). The correlation between the apparent fidelity and the 
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classroom performance on the expository component of the STAAR fourth-grade writing was moderate and 
negative (-0.54). As fidelity increased based on classroom observations, students’ performance on the 
STAAR declined based on the classroom observations. Observations of what actually happened during the 
delivery of the lessons may shed more light on the relationship between fidelity and performance using 
Writer's Workshop at the fourth-grade in HISD. The observations were conducted by only one observer. 


The difference in the percentage of fourth-grade students whose teachers completed or were reported as 
“no show” in the workshop and who met Level Il: Satisfactory at the 2016 progression standard was 
negligible. However, 4.8 percent more students whose teachers were reported as “no show” compared to 
the students whose teacher completed the workshop met Level III: Advanced standard on the fourth-grade 
STAAR writing test. STATA treatment effects (teffects) with regression adjustment (ra) showed that students 
whose teachers did not participate in the Writers Workshop PD training had an average scale score 
(POmean) of 3823.9 while those whose teachers participated had an average scale score that was 104.8 
scale score points lower than the POmean (3719.1). The difference was statistically significant (p<.001). 
Similarly, students whose teachers completed the workshop received a lower number of items correct on 
the expository component of the STAAR writing test compared to their peers whose teachers did not 
participate in the workshop. Here too, the difference was statistically significant (p=.001). A possible 
explanation for the difference in the STAAR performance between the treatment and non-treatment groups 
may be the extent to which the Writer’s Workshop was implemented and its key protocols were adhered to. 
More than half of the teacher respondents devoted 30 minutes to instruction in writing, which is 15-30 
minutes less than the recommended instruction time for delivering Writers Workshop. Subsequently, most 
students may have had insufficient time to write, which could mean they may have received inadequate 
feedback and insufficient time to demonstrate improvements in their writing. This time disparity could have 
been due to the departmentalization of instruction in those schools which resulted in reduced teaching time 
to 30-minute periods. 


Recommendations 


e Teacher completion of the professional development designed to enhance preparation for effective 
delivery of the Writer's Workshop needs to be encouraged. The high contract to participation cost 
incurred in delivering this professional development and the failure of teacher participants to confidently 
confirm students’ ability to outline the writing process using the Bomer’s Flowchart makes the case for 
the encouragement. 


e Students and teachers may require additional resources to effectively teach writing at the elementary 
school level. It may be essential to ensure that writing receives the same level of attention at all 
elementary grade levels in addition to the fourth grade where writing is tested. 


e Future evaluations should focus on observing the delivery of the Writer’s Workshop, particularly before 
the testing period, and possibly multiple times during the school year to assess teachers’ actual writing 
instructional practices. This would allow for a more thorough observation of the Writer's Workshop and 
a more robust assessment of fidelity. 
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Appendix a 
Rating scale 


Rating scale question calculates a weighted average based on the weight assigned to each answer choice. 
The average rating is calculated as follows, where: 


w = weight of answer choice 
X = response count for answer choice 


XiWi_+ X2We + X3Ws_... XnWh 
Total 


SurveyMonkey (1998-2016). Rating and Ranking Average Calculations. Retrieved from: 
http://help.surveymonkey.com/articles/en_US/kb/What-is-the-Rating-Average-and-how-is-it-calculated 
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Appendix B 


Writers Workshop classroom observation results showing fidelity, 2015-2016 


% 
Observation Prompts Koawc K04GS | KO4CR | HO5BJ | HO4BK | FO04GS | POSMR | PO5AA P04GR PO4SK | PO4NC Satisfy 
(Fidelity Criteria) [event 
Writing visuals — charts, posters, Yes Yes Yes No Yes Yes No No Yes Yes Yes 72.7 
etc. 
Writer’s notebooks — dated 
entries, skipped lines, pencil Yes Yes Yes No Yes Yes No No Yes Yes Yes 72.7 


writing, (What writers do - Think, 
Draw, Label, Write, Add details) 
Mini-Lessons (15mins.): Gather 
students, review expectations 

(SLANT), Revisit read aloud text, Yes Yes Yes No No No Yes No Yes No No 45.5 
Use anchor chart, 


Independent writing (12mins.): 
return students to desks, 
independent writing, and select 
students to share. Yes Yes Yes No No Yes No Yes Yes Yes Yes 63.6 


Sharing (8mins.): Gather 
students, select students share 
writing, commend, praise Yes Yes Yes No No Yes No Yes No No Yes 54.5 


Writing Process 
Prewriting/Brainstorming and 


Planning — graphic — organizers Yes Yes Yes Yes No Yes Yes Yes No Yes Yes 81.8 
Draft Writing Yes Yes Yes Yes No Yes No Yes No Yes Yes 72.7 
Read, Respond and Revise — Yes Yes Yes Yes No Yes No No No Yes Yes 63.6 
reading written work 

Teacher Yes Yes Yes Yes No Yes No Yes No Yes Yes 72.7 


Conference/Responding 
Revise and Edit — return draft 
(ARMS - add, remove, move, 
substitute) (CUPS- Yes Yes Yes Yes No Yes No Yes No Yes Yes 72.7 
capitalization, usage, 
punctuation, spelling) 


Proofread — peer review (oral or Yes Yes Yes Yes No Yes No No No Yes Yes 63.6 
written) and teacher review 
Publish (Sample writing located Yes Yes Yes No No No No No No Yes Yes 45.5 


in the room/ dossier/online) 
Use of Texts 

Mentor text — student-selected Yes Yes Yes No No Yes No No No Yes Yes 54.5 
reference text for modeling 
Listening Habits (SLANT) 


Sit up straight Yes Yes No No No Yes No No No No No 18.2 
Listen Yes Yes No No No Yes No Yes No No No 27.2 
Ask and Answer Yes Yes No No No Yes No Yes No No No 36.4 
Nod your head No Yes No No No Yes No No No No No 18.2 
Track the speaker No Yes No No No Yes No Yes No No No 27.2 
Other Writing Features 

Good habits of writers (charts, No No Yes No No Yes No No No Yes Yes 36.4 
oral, posters, etc.) 

Develop writing ideas (pictures, No No No No No Yes No Yes No Yes Yes 36.4 
objects, drawings) 

TEKS-Writing lesson link Yes Yes Yes Yes No Yes No No No Yes Yes 63.6 
Heart Maps (Charts, posters, No No Yes No No No No No No Yes Yes 27.2 
notebooks) 

Celebration artifacts (Author's 

Chair, sharing) celebration board No Yes Yes No No No No Yes No Yes Yes 45.5 
Writing Assessment or progress No Yes Yes No No Yes No No No Yes Yes 45.5 
chart 

Apparent Fidelity 16 21 17 7 2 20 1 11 4 17 18 

% Fidelity 66.7 87.5 70.8 29.1 ees) 83.3 4.2 45.8 16.7 70.8 75.0 

% Met Level Il: Satisfactory 74.8 66.9 29.9 58.5 63.6 87.9 52.7 77.8 

Mean Raw Score on 

Composition Piece 3.68 3.34 3.28 3:53 3.63 5.39 3.48 4.02 
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